Measuring Air Traffic Controller Performance in a High-Fidelity Simulation

In the summer of 1997, the Air Traffic Selection and Training (AT-SAT) High Fidelity Simulation Study was conducted at the FAA Academy in Oklahoma City, OK. The purpose of the study was to test the performance of 107 operational en route controllers during 2½ days of simulations. The performance of these controllers during the high-fidelity simulations was compared with their performance on two medium-fidelity performance measures to assess the construct validity of the latter measures to serve as criteria against which to validate a set of selection tests. The reports included in this document describe the high-fidelity simulation exercise, the development of performance measures utilized during the exercise, and the interrelationships between the performance measures. The first report describes the development of a work sample approach to capturing air traffic controller performance, and establishes that high fidelity performance measures can adequately reflect the performance of the controller. The work sample was developed in an environment that simulated as nearly as possible the actual conditions existing in the controller's job, but was conducted in a " generic " airspace. Scenario development included the most important tasks from the task-based job analysis developed for the AT-SAT project. Sufficient time was provided for participating controllers to learn the airspace and procedures and demonstrate their knowledge through 1) a multiple choice test of airspace knowledge and 2) running 8 practice scenarios. Performance was measured by 1) an over-the-shoulder (OTS) rating scale, 2) counts of mistakes, 3) counts of actions that would be required to move aircraft from the sector at the end of the scenario, and 4) statistics derived from aircraft positions and controller/pilot data entries recorded for the simulation. The second report used measures collected during the high-fidelity simulation study to predict the overall OTS performance rating. It was found that a model that included both counts of mistakes and the computer-derived performance measures predicted the OTS rating reasonably well, while a model containing only the computer-derived measures did not. Remaining actions did not contribute to the prediction of the OTS rating in addition to the contribution provided by the other types of measures. Henry Mogilka Federal Aviation Administration FAA Academy controllers. These include full-scale dynamic simulations that allow controllers to direct the activities of a sample of simulated air traffic, performing characteristic functions such as ordering changes in aircraft speed or flight path, all within a relatively standardized work sample framework. The …